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ABSTRACT 

Objectives: Specialist early intervention services (EIS) 
for people aged 14-35 years with first episodes of 
psychosis (FEP) have been commissioned throughout 
England since 2001. A single estimate of population 
need was used everywhere, but true incidence varies 
enormously according to sociodemographic factors. 
We sought to develop a realistically complex, 
population-based prediction tool for FEP, based on 
precise estimates of epidemiological risk. 
Design and participants: Data from 1037 participants 
in two cross-sectional population-based FEP studies were 
fitted to several negative binomial regression models to 
estimate risk coefficients across combinations of different 
sociodemographic and socioenvironmental factors. We 
applied these coefficients to the population at-risk of a 
third, socioeconomically different region to predict 
expected caseload over 2.5 years, where the observed 
rates of ICD-10 F10-39 FEP had been concurrently 
ascertained via EIS. 

Setting: Empirical population-based epidemiological data 
from London, Nottingham and Bristol predicted counts in 
the population at-risk in the East Anglia region of England. 
IVIain outcome measures: Observed counts were 
compared with predicted counts (with 95% prediction 
intervals (PI)) at EIS and local authority district (LAD) 
levels in East Anglia to establish the predictive validity of 
each model. 

Results: A model with age, sex, ethnicity and population 
density performed most strongly, predicting 508 FEP 
participants in EIS in East Anglia (95% PI 459, 559), 
compared with 522 observed participants. This model 
predicted correctly in 5/6 EIS and 19/21 LADs. All models 
performed better than the current gold standard for 
EIS commissioning in England (716 cases; 95% PI 
664-769). 

Conclusions: We have developed a prediction tool for 
the incidence of psychotic disorders in England and 
Wales, made freely available online (http://www. 
psymaptic.org), to provide healthcare commissioners with 
accurate forecasts of FEP based on robust epidemiology 
and anticipated local population need. The initial 
assessment of some people who do not require 
subsequent EIS care means additional service resources, 
not addressed here, will be required. 



ARTICLE SUMMARY 



Article focus 

■ Commissioners require precise information on 
the health needs of their local populations to 
effectively plan health services. 

■ A failure to arm mental health commissioners 
I with precise epidemiological data led to misesti 
' mation of actual activity in early intervention in 

psychosis services (EIS). 

■ We sought to develop a prediction tool for the 
incidence of first episode psychosis (FEP), by 

I applying precise estimates of epidemiological 
risk in various sociodemographic groups to the 
structure of the population at-risk in a different 
region, where the observed incidence had been 
concurrently ascertained. 

Key messages 

■ A model of psychosis incidence which included 
age, sex, ethnicity and population density yielded 
precise FEP predictions in our new region, out- 
performing the Department of Health ir 
England's current gold standard for EIS 
commissioning. 

■ While our model provides forecasts of the 
burden of FEP in different populations, the initial 
assessment of some people who do not require 
subsequent EIS care means additional service 
resources, not addressed here, will be required. 

■ We have translated this model into a freely avail 
able prediction tool (http://www.psymaptic.org) 
to facilitate evidence-based healthcare commis- 
sioning of socioculturally relevant services 
according to local need. 



BACKGROUND 

Commissioners of health and social care 
require precise information on the health 
needs of their local populations/ especially if 
parity of mental and physical health is to be 
realised.^ Mental health disorders alone rep- 
resent the leading disease burden in the UK 
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ARTICLE SUIVIIVIARY 



Strengths and limitations of this study 

■ Our modelling approach used robust epidemiological data from 
two large studies of first episode psychosis in England to 
provide estimates of incidence in a third study region, produ- 
cing accurate FEP forecasts. 

■ While our models provide estimates of the expected clinical 
burden of FEP in the community, services may see a broader 
range of psychopathology consuming resources, or incepted 
rates may be influenced by supply-side organisational factors. 

■ Owing to data availability, it was not possible to validate our 
prediction tool in settings outside of England and Wales, or for 
specific psychotic disorders. As data become available, we will 
extend the capability of our prediction tool, including into 

^ other settings and disorders. 



(22.8%).^ They contribute substantially to healthcare 
expenditure and societal costs even before physical ill 
health is taken into account. The Centre for Mental 
Health estimated the total costs of mental health to 
British health services and society at £105 billion in 
2009/2010,^ a figure expected to double over the next 
20 years. ^ These are serious challenges compounded by 
a paucity of information on which to commission appro- 
priate services. Early intervention in psychosis services 
(EIS) for people aged 14-35 years with a first episode of 
psychosis (FEP) offer a useful example of failure to map 
services to local need. 

EIS are a major evidence-based innovation, systematic- 
ally commissioned throughout England and Wales over 
the past decade.^ When EIS intervention is sustained, 
there is evidence that people with psychosis achieve 
better functional and social outcomes.^ ^ Such services 
are also highly cost-effective.^ ^ ^ However, EIS were ori- 
ginally commissioned on an anticipated rate of 150 new 
cases of any psychotic disorder per 1 000 000 of the total 
population per year in the Department of Health's 
Mental Health Policy Implementation Guide 
(MH-PIG).^ In 2001 in England and Wales, 29.3% of the 
population were aged 14-35 years, meaning that the 
MH-PIG commissioned incidence rate was approxi- 
mately 51 cases/ 100 000 person-years in the age range 
covered by EIS. Following their deployment, anecdotal 
reports began to emerge from EIS in different regions 
to suggest that a uniform figure for commissioning was 
simultaneously underestimating^^ and overestimating^^ 
the actual observed need in urban and rural popula- 
tions, respectively. Recent epidemiological evidence of 
FEP incidence in rural communities in England has sug- 
gested that rates are somewhat lower than the uniform 
figure upon which services were commissioned,^^ con- 
firming previous calls that a 'one-size-fits-all' prescription 
for EIS implementation is unlikely to lead to the effi- 
cient allocation of finite mental health resources. 

Using rich epidemiological data on variation in the 
incidence of FEP according to major sociodemographic 



risk factors 



16-19 



we describe the development and 



validation of a population-level prediction tool capable 
of accurately estimating the expected incidence of psy- 
chiatric disorder, based on the sociodemographic struc- 
ture of the population in a given region. Applied to FEP 
as proof-of-concept, we show that it is possible to closely 
predict the expected incidence in a given population, 
where the observed count of cases was within the predic- 
tion intervals (PI) forecast by our models. We applied 
our most precise prediction model to the population of 
England and Wales to provide health commissioners 
with a translational epidemiological prediction tool to 
underpin information-based service planning. 

METHODS 

Our prediction models were based on epidemiological 
data from the Aetiology and Ethnicity in Schizophrenia 
and Other Psychoses (^SOP) and the East London First 
Episode Psychoses (ELFEP) studies, two methodo- 
logically similar population-based FEP studies. We fitted 
various count-based regression models with different 
combinations of sociodemographic and socioenviron- 
mental factors, well established in the literature to be 
associated with the incidence of psychotic disorder.^^ 
We first established the relative apparent validity of each 
model by estimating model-fit diagnostics to assess how 
well each model fitted the empirical data (henceforth, 
the prediction sample). We next sought to estimate the 
external validity of each model by applying model-based 
parameter coefficients to the population structure of a 
purposefully different region of England, East Anglia 
(henceforth, the validation sample). This out-of-sample 
prediction technique allowed us to obtain the expected 
incidence of disorder in this region forecast by each 
model, which we compared with observed rates simul- 
taneously ascertained in this region via the ongoing 
Social Epidemiology of Psychoses in East Anglia 
(SEPEA) study.^^ We performed various model-fit diag- 
nostics to identify which, if any, model demonstrated 
utilisable predictive capability. 

Empirical data underlying prediction models (prediction 
sampie) 

Case ascertainnnent (nunnerator) 

The designs of the ^SOP and ELFEP studies have been 
described in detail elsewhere, with features relevant 
to the present paper summarised here. Case ascertain- 
ment took place over 2 years in ELFEP (Newham: 1996- 
1998; Tower Hamlets 8c Hackney: 1998-2000) and the 
Southeast London and Nottingham centres of the 
^SOP study (1997-1999), and over the first 9 months of 
1997 in Bristol (yESOP). All service bases were screened 
regularly for potential new contacts aged 16-64 years 
(18-64 in ELFEP) resident within these catchment 
areas. Leakage studies were conducted to identify parti- 
cipants missed by this initial screen, but meeting inclu- 
sion criteria for FEP.^^ All participants who received 
an ICD-10 FlO-39 diagnosis for psychotic disorder 
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following assessment via the Schedules for Clinical 
Assessment in Neuropsychiatry were included in the inci- 
dent sample, except those with an organic medical basis 
to their disorder or profound learning difficulty. Data 
on age-at-contact, sex and ethnicity were collected on 
included participants. We geocoded participants' resi- 
dential postcode at first contact to their corresponding 
local authority district (LAD) to allow us to model pos- 
sible neighbourhood effects associated with the inci- 
dence of psychotic disorder, such as population density 
or socioeconomic deprivation. 

Population at-risk 

We estimated the population at-risk using the 2001 
Census of Great Britain, adjusted for study duration, and 
stratified by age group (16-17, 18-19, then 5-year age 
bands), sex and ethnicity. Ethnicity was based on self- 
ascription according to 1 of 10 categories derived from 
the census: white British, non-British white, black 
Caribbean, black African, Indian, Pakistani, Bangladeshi, 
mixed white and black Caribbean, other mixed ethnic 
backgrounds and all other ethnicities. 

Socioenvironmental variable estimation 

We estimated LAD-level deprivation using the 2004 
Index of Multiple Deprivation (IMD) in England, which 
estimated domains of deprivation using measures pre- 
dominantly collected close to the time of our case 



Table 1 Description of included socioenvironmental 
variables*t 



Variable 



Classification and description 



Multiple 
deprivation 



I 



Extent of 
deprivation 

Income 
deprivation 
Employment 
deprivation 

Population density 



Weighted data from routine national 
sources across seven domains: ^ 
income, employment, education, 
health, barriers to housing and 
services, living environment, crime. 
Continuous, z-standardised scores for 
analysis 

Proportion of LAD population living in 
20% most deprived SOA in England 
(%) 

Proportion of all people in LAD 
classified as income deprived (%) 
Proportion of adults of working age in 
LAD classified as employment 
deprived (%) 

Population density at LAD level 
(people per hectare) 



* Prediction sample sources: Population density — 2001 census 
estimates; deprivation variables: 2004 Indices of Deprivation, 
predominantly collected from data sources close to /ESOP and 
ELFEP case ascertainment periods (ie, 1997-2000). 
-[Validation sample sources: Population density — 2009 mid-year 
census estimates; deprivation variables: 2010 Indices of 
Deprivation, predominantly collected from data sources just prior 
to the SEPEA case ascertainment period (2008). 
IMD, Index of Multiple Deprivation; LAD, local authority district; 
SOA, super output area. 



ascertainment periods (see table 1).^^ We z-standardised 
English LAD IMD scores to have a mean of zero and SD 
of 1, and extracted IMD z-scores for the 14 LADs in the 
yESOP and ELFEP studies. To inspect whether any par- 
ticular deprivation domain was a better predictor of 
psychosis incidence than IMD, we also considered 
LAD-level income deprivation, employment deprivation 
and the extent of deprivation in our models (table 1). 
We estimated population density by dividing each LAD's 
usual resident population by its area (in hectares) , using 
ArcGIS V9.3 software. 

Observed data for external validation of prediction models 
(validation sampie) 

Observed participants and population at-risk data for 
our validation sample were obtained from the SEPEA 
study, an ongoing study of the incidence of psychotic dis- 
orders incepted over 3.5 years (2009-2013) through one 
of six EIS covering 20 LADs and a subsection of 1 LAD 
(the town of Royston, Hertfordshire) in Norfolk (three 
EIS: West Norfolk, Central Norfolk, and Great Yarmouth 
and Waveney), Suffolk (one EIS) and Cambridgeshire, 
Royston and Peterborough (CAMEO North and South 
EIS).^^ 

Case ascertainment 

To establish the incepted incidence of FEP as seen 
through EIS, entry criteria for the SEPEA study were: 

► Referral to an EIS in East Anglia for a suspected first 
episode of psychosis; 

► Aged 16-35 years at first referral to EIS (17-35 years 
in CAMEO services) ; 

► Resident within the catchment area at first referral; 

► First referral during case ascertainment period 
(2009-2013). 

At 6 months after EIS acceptance, or discharge from 
the service, whichever was sooner, we asked the clinician 
responsible for care to provide an ICD-10 FlO-39 psychi- 
atric diagnosis using all information available. We 
excluded participants without a clinical FEP diagnosis, 
or participants presenting with an organic basis to their 
disorder or profound learning disability. For the remain- 
ing participants, basic sociodemographic and postcode 
information was recorded and classified in the same way 
as in the prediction sample. We included participants pre- 
senting to EIS during the first 2.5 years of the ongoing 
SEPEA study. 

Population at-risl< 

We estimated the population at-risk of East Anglia using 
2009 mid-year census estimates published by the Office 
for National Statistics (ONS) at the LAD level, by age 
group, sex and ethnicity.^^ These estimates used the 
2001 census base, adjusted for immigration, births and 
deaths each year. It was not possible to obtain 2009 
mid-year estimates for the town of Royston, because data 
were only published at LAD level. Here, we used 
denominator data from the 2001 census data in order to 
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estimate the population at-risk in Royston. We do not 
believe that this would have substantially invalidated our 
results as this town represented 0.6% of the overall 
population at-risk (n=9555) in the SEPEA study 
Denominator data were multiplied by 2.5 to account for 
person-years of exposure in the validation sample. 

Socioenvironmental variable estimation 

For each LAD in the SEPEA study, we obtained corre- 
sponding socioenvironmental variables to those 
included in our prediction sample, using updated data col- 
lected as close to the SEPEA case ascertainment period 
as possible. Population density was estimated using 2009 
mid-term population estimates. Our measures of depriv- 
ation were derived from IMD 2010,^^ which was esti- 
mated in an analogous way to 2004 data, but collected 
from sources obtained immediately prior to the SEPEA 
study. 

Statistical tecliniques 

Dataset generation 

We constructed a dataset for the regression analysis of 
count data by pooling data from the ^SOP and ELFEP 
studies (the prediction sample). Data were stratified by age 
group, sex, ethnicity and LAD, such that each stratum 
(N=2536) represented the total count of FEP cases in a 
unique sociodemographic group for a given LAD, with a 
corresponding estimate of the population at-risk, treated 
as an offset in our models. Our socioenvironmental 
measures (population density, deprivation) were 
adjoined to the dataset for each LAD. Population at-risk 
data from the validation sample were stratified in the 
same way and retained in a separate database. Here, the 
count of cases, which we wished to predict, was entered 
as a vector of missing data which would be populated 
with predicted case estimates following prediction 
modelling. 

Prediction models 

We used the prediction sample data to fit negative bino- 
mial regression models to obtain parameter coefficients 
of incidence for the sociodemographic and socioenvir- 
onmental factors included in each model. We consid- 
ered the internal and external predictive capabilities of 
six models, all of which contained age group, sex, an 
age-sex interaction term and ethnicity. Model 1 con- 
tained no further covariates. Model 2 also included 
IMD. We replaced IMD with either income, employment 
or the extent of deprivation, respectively, in models 3-5. 
Model 6 included population density. Initial exploration 
of the prediction sample data indicated the presence of 
possible overdispersion (variance (5^=1.37) exceeded 
mean ()i=0.4) count of cases), so negative binomial 
regression was preferred to Poisson regression since it 
explicitly models any overdispersion with an extra disper- 
sion parameter. 



Apparent model validity and prediction 

We assessed apparent model validity in three ways. First, 
we used Akaike's Information Criterion (AIC) to assess 
the respective overall fit of each model to the data. 
Second, we conducted K-fold cross-validation to assess 
each model's apparent validity to predict cases within 
the prediction sample. This method randomly allocated 
strata in the prediction sample into K subsets. Each model 
was then re-estimated on K-l subsets (the training data) 
to predict the expected counts of cases in the Kth subset 
(the test data). This was repeated over K trials, such that 
each stratum in the dataset appeared exactly once as the 
test data. At the end of this process, we derived Lin's con- 
cordance correlation coefficient (CCC) and 95% CI to 
estimate the correlation between the predicted and 
observed counts of cases across all strata in the prediction 
sample. Finally, we estimated the root mean squared 
error (RMSE) to determine the average error between 
fitted and observed values from each model. Lower 
RMSE scores indicated a smaller prediction error. The 
RMSE is derived as 

RMSE =yEiKZlZ 

where yi and are the observed and predicted counts 
of cases in the ith stratum, respectively, and n is the 
number of strata. 

We repeated K-fold cross-validation h times, generating 
K new random divisions of the data each time. We 
retained model-fit diagnostics across Kh iterations, and 
reported the mean of Lin's CCC and RMSE to provide 
summary cross-validation statistics for each model. We 
specified K=10 and h=20, as recommended for cross- 
validation to obtain precise model-fit diagnostics.^^ 

External model prediction and validation 

We retained parameter coefficients from each model 
(using the full prediction sample data) and applied these to 
the corresponding population at-risk in the validation 
sample dataset. This gave out-of-sample prediction esti- 
mates for the expected count of cases in each stratum of 
the validation sample, given the model. We summed 
expected counts across relevant strata to estimate the 

(1) total predicted count of cases in the SEPEA region, 

(2) predicted counts in each EIS and (3) predicted 
counts by LAD. These counts were further stratified by 
broad age group (16-35, 36-64 and 16-64 years). 
Because the census (denominator) data were unavailable 
for 35-year-olds alone (needed to estimate their contribu- 
tion to predicted counts in the age range for EIS, 
16-35 years), we assumed that the risk coefficient was the 
same across all ages within the 35-39-year-old age group. 
We apportioned predicted counts on a 1:4 ratio (35:36- 
39 years) to their respective broad age groups. 

To determine how well the MH-PIG^ figure of 51 new 
cases per 100 000 person-years for EIS performed as a 
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predictive tool, we also estimated the predicted count of 
cases in the validation sample under this scenario, which 
we termed 'Model 7'. 

We derived 95% Pis for all summary predictions from 
first principles, since their derivation is not straightfor- 
ward, nor routinely implemented by statistical software. 
Pis are similar to CIs, but account for SEs introduced in 
both the prediction and validation samples. We developed 
a bootstrap-like approach to obtain Pis from each model 
by simulating 1000 model-based realisations of the quan- 
tities we wished to predict, where we took the para- 
meters to be the maximum likelihood estimates. We 
obtained the lower and upper bounds of the Pis as the 
corresponding quantiles of the simulated realisations 
(see appendix for full details). 

To assess each model's external predictive capabilities, 
we considered five markers of predictive accuracy. We com- 
pared the number of times the observed count of cases in 
the SEPEA study fell within the Pis estimated from each 
model for (1) the SEPEA region, (2) at the EIS level and 
(3) at the LAD level. We also derived ElS-level (4) and 
LAD-level (5) RMSE scores to estimate prediction error 
from each model in our validation sample. We ranked 
model performance (1: best and 7: worst) on these five 
measures, and estimated an overall mean rank to deter- 
mine the overall predictive validity of each model. 

Observational data on first episode psychosis in our 
validation sample were not available for the age range 
36-64 years, so external validation was restricted to the 
16-35 year old age range. For completeness, however, we 
also reported the overall predicted count of cases for 
this age group from each model. 

Extrapolation to the UK 

Guided by our validation procedures, we identified which 
model had the greatest overall predictive validity, and pro- 
posed this as a candidate for FEP incidence prediction in 
England and Wales. We repeated out-of-sample prediction 
on the sociodemographic and socioenvironmental popula- 
tion characteristics of each LAD in England and Wales to 
obtain national-level and LAD-level predictions. 
Denominator data were obtained from the ONS 2009 
mid-term estimates and stratified as previously described. 
Overall counts were derived for three broad age groups 
(16-35, 36-64 and 16-64 years), and for each of these, by 
sex and ethnicity. The 95% Pis were estimated as before. 
We visualised these data on maps and in tables to provide 
healthcare planners and commissioners with an easy-to-use 
tool to forecast the expected incidence of psychotic dis- 
order in England and Wales. We have made this available 
as a free, open-use prediction tool, known as PsyMaptic 
(V0.5) (Psychiatric Mapping Translating Innovations into 
Care; http://www.psymaptic.org). Counts of cases pre- 
dicted by our model were compared with those obtained 
under the Department of Health's uniform rate in each 
LAD. We expressed these comparisons as ratios with 95% 
CIs derived using the same method as for standardised 
morbidity ratios (SMR). This approach was conservative 



because here we substituted the usual numerator in an 
SMR, the observed, O, for a predicted count. Unlike an 
observed count, no sampling variation is present for the 
predicted count, only uncertainty due to the model from 
which the prediction was estimated. Since variance in the 
prediction is therefore much smaller than the variance nor- 
mally present for the numerator (O), this led to conserva- 
tive estimates of 95% CI. Ratios in LAD where 95% CI did 
not span unit)^ could therefore be interpreted as regions 
where there was strong evidence that the predictions from 
our model differed significantly from those predicted by 
the department of health's uniform rate. 

Software 

All negative binomial regression models, out-of-sample 
prediction and estimation of 95% PI were conducted in 
R (V2.15.1). Cross-validation and model-fit diagnostics 
were conducted in Stata (V.ll). Prediction maps for 
England and Wales were created using StatPlanet Plus 
(V.3.0) visualisation software.^ 

RESULTS 
Prediction sample 

Our prediction models contained data on 1037 persons 
with a first episode psychosis in the yESOP (n=553; 
53.3%) and ELFEP (n=484; 46.7%) studies, ascertained 
from over 2.4 m person-years at-risk. Twelve participants 
were excluded from the original tESOP sample because 
they were of no fixed abode and could not be geocoded 
to an LAD.^^ 

The population at-risk in the prediction sample came 
from LADs with higher median levels of multiple and 
employment deprivation, extent of deprivation and 
population density than the population at-risk in the val- 
idation sample, though there were no statistically signifi- 
cant differences in median income deprivation between 
the two samples (see online supplementary table SI). 

Parameter coefficients obtained from the full predic- 
tion sample following negative binomial regression are 
shown in table 2. As previously reported from these 
data,^^ incidence rates were generally raised in ethnic 
minority groups compared with the white British popula- 
tion. Models 2-6 included a measure of LAD depriv- 
ation (models 2-5) or population density (model 6), 
which were all significantly associated with an increased 
incidence of psychotic disorder, after control for 
individual-level confounders. Each of these models pro- 
duced a lower AIC score than a model fitted solely with 
individual-level covariates (model 1), indicating a better 
fit. Cross-validation suggested that all models achieved 
good CCC agreement between predicted and observed 
cases, with low RMSE values (table 2). 

Validation sample 

Observed participants 

We identified 572 potential participants over the first 
30 months of the SEPEA study, aged 16-35 years, who 
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Table 2 Prediction models, covariates and fit: all clinically relevant psychoses (F10-39) 





MnHpl 1 


MnHpl 7 


MnHpl 


MnHpl ^ 


MnHpl 


MnHpl R 




IRR ^95% Ch 

II 11 1 t«/w /O ^^1/ 


IRR ^95% Ch 

II 11 1 t«/w /O ^^1/ 


IRR ^95% Ch 

II 11 1 t«/w /O ^^1/ 


IRR ^95% Ch 

II 11 1 t«/w /O ^^l| 


IRR ^95% Ch 

II 11 1 t«/w /O ^^'/ 


IRR ^95% Ch 

II 11 1 t«/w /O ^^1/ 


Age groupxsex interaction* 


p=0.07 


p=0.06 


p=0.06 


p=0.06 


p=0.06 


p=0.06 


Ethnicity 














White British 


1 


1 


1 


1 


1 


1 


Non-British white 


2.0 (1.5 to 2.5) 


1.7 (1.4 to 2.2) 


1.7 (1.3 to 2.2) 


1.7 (1.3 to 2.2) 


1.8 (1.4 to 2.3) 


1.7 (1.3 to 2.2) 


Black Caribbean 


6.0 (4.9 to 7.3) 


5.3 (4.3 to 6.5) 


5.2 (4.3 to 6.4) 


5.2 (4.3 to 6.4) 


5.4 (4.5 to 6.6) 


5.1 (4.2 to 6.3) 


Black African 


4.1 (3.3 to 5.1) 


3.6 (2.9 to 4.5) 


3.5 (2.8 to 4.4) 


3.5 (2.8 to 4.4) 


3.7 (3.0 to 4.6) 


3.5 (2.8 to 4.3) 


Indian 


1.7 (1.2 to 2.5) 


1.5 (1.1 to 2.2) 


1.5 (1.0 to 2.2) 


1.5 (1.0 to 2.1) 


1.6 (1.1 to 2.3) 


1.6 (1.1 to 2.2) 


Pakistani 


1.8 (1.2 to 2.5) 


1.6 (1.0 to 2.5) 


1.6 (1.0 to 2.4) 


1.6 (1.0 to 2.4) 


1.6 (1.1 to 2.5) 


1.8 (1.3 to 2.7) 


Bangladeshi 


2.1 (1.5 to 2.8) 


1.7 (1.2 to 2.3) 


1.7 (1.2 to 2.3) 


1.6 (1.2 to 2.2) 


1.8 (1.3 to 2.5) 


1.8 (1.4 to 2.4) 


Mixed white and black 


4.3 (2.8 to 6.7) 


3.9 (2.5 to 6.0) 


3.9 (2.5 to 6.0) 


3.9 (2.5 to 6.0) 


4.0 (2.6 to 6.1) 


3.9 (2.5 to 6.1) 


Caribbean 














Mixed, other ethnicities 


1.3 (0.8 to 2.3) 


1.2 (0.7 to 2.1) 


1.2 (0.7 to 2.1) 


1.2 (0.7 to 2.1) 


1.2 (0.7 to 2.1) 


1.2 (0.7 to 2.1) 


Other ethnicities 


2.2 (1 .6 to 3.0) 


1.9 (1.4 to 2.7) 


1.9 (1.4 to 2.6) 


1.9 (1.4 to 2.6) 


2.0 (1.4 to 2.7) 


1.9 (1.4 to 2.7) 


Socioenvironmental variables 














IMD (z-score) 


— 


1.184 (1.101 to 
1 .274) 


— 


— 


— 


— 


Extent of deprivation (%) 


- 


- 


1 .008 (1 .004 to 
1.011) 


- 


- 


- 


Income deprivation (%) 








1.025 (1.015 to 
1.035) 






Employment deprivation (%) 










1.062 (1.032 to 
1.093) 




Population density (pph) 












1.005 (1.003 to 
1 .007) 


Model-fit diagnostics 














AlCt 


2571.8 


2552.4 


2551.3 


2549.6 


2556.4 


2556.3 


Mean Lin's CCC (95% C\)t 


0.75 (0.74 to 0.77) 


0.77 (0.75 to 0.78) 


0.77 (0.75 to 0.79) 


0.77 (0.75 to 0.78) 


0.77 (0.75 to 0.78) 


0.76 (0.74 to 0.77) 


Mean RMSE (SD)§ 


0.75 (0.11) 


0.74 (0.11) 


0.74 (0.10) 


0.74 (0.10) 


0.74 (0.11) 


0.76 (0.13) 



*AII models fitted with age group by sex interaction given a priori evidence for effect modification.^® Likelihood ratio test p values reported between models with and without an interaction term 
fitted between age group and sex. Specific IRR has not been reported for clarity, but is available on request. 
tLower scores denote improved model fit. 

ilHigher scores indicate greater correlation between observed and predicted count of cases in the prediction sample. Mean CCC and 95%CI reported following h=20 trials during 
cross-validation. 

§Lower scores indicate lower prediction error. Mean RMSE and SD reported following h=20 repeats of K-fold cross-validation, where K=10. 
AlC, Akaike's Information Criterion; CCC, Lin's correlation concordance coefficient; IRR, incidence rate ratio; RMSE, root mean squared error.. 
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met initial acceptance criteria for EIS in East Anglia. We 
excluded 50 participants (8.7%) who did not meet clin- 
ical criteria for the ICD-10 psychotic disorder. This left 
an incidence sample of 522 participants from nearly 
1.4 m person-years at-risk (37.4/100 000 person-years; 
95% CI 34.3 to 40.7). A further 2.3 m person-years 
at-risk accrued in the same region for people aged 36- 
64 years over this period. Median levels of multiple, 
income and employment deprivation in the region did 
not differ significantly from the remainder of England, 
although the median population density and extent of 
deprivation in East Anglia were lower than elsewhere in 
England (see online supplementary table SI). 

External model prediction and validation 

The overall observed count of cases, aged 16-35 years, 
in the validation sample (n=522) fell within 95% Pis in 
four of seven models (models 3-6, table 3). Of these, 
the observed count (n=522) was closest to the point esti- 
mate for model 6 (508.5; 95% PI 449.0, 559.0), fitted 
with age group, sex, their interaction, ethnic group and 
LAD population density. The observed count of cases 
also fell within Pis from this model in five of six EIS in 
the study region, and 19 of 21 LADs, the most in any 
model (table 4). This model had the lowest error scores 
at the EIS (RMSE=11.6) and LAD (RMSE=6.1) levels of 
any model. Overall, model 6 was ranked highest across 
all external model-fit diagnostics (table 4). All models 
outperformed the department of health's uniform 
figure of 51 per 100 000 person-years (model 7), which 
generally overestimated cases in the validation sample 
(overall prediction: 715.7 cases; 95% PI 664.0, 769.0). 

We reported predicted cases aged 36-64 years from 
our models (table 4), although we could not test these 
in the validation sample. Model 6 predicted an additional 
262.9 cases aged 36-64 years over a 2.5-year period in 
East Anglia (95% PI 233.0, 297.0). 

We inspected the stratum-specific external validity of 
our best-fitting model (model 6, see online supplemen- 
tary table S2), which performed accurately for sex- 
specific predictions, but less well in age-specific and 
ethnicity-specific strata. Thus, our model tended to 
underpredict observed cases in people aged 16-19 years, 
but overpredicted cases observed in people over 25 years 
old. With respect to ethnicity, model predictions were 
consistent with observed FEP cases for people of 
non-British white, black African, Bangladeshi and mixed 
ethnicities. However, our model tended to underpredict 
observed rates in the white British group, and overpre- 
dicted rates in the black Caribbean, Indian and 
Pakistani populations. 

Extrapolation to England and Wales 

We predicted the expected count and incidence of first 
episode psychosis per annum in each LAD in England 
and Wales based on model 6, and visualised these data 
in maps and tables freely available at www.psymaptic.org. 
Many maps can be visualised (eg, see online 



supplementary figure SI), including the overall pre- 
dicted incidence counts and rates for each broad age 
group at the LAD level, and by sex. We will make 
PsyMaptic data available by ethnic group when we can 
improve the validity in ethnic-specific strata. According 
to our model, the annual number of new FEP cases in 
England and Wales would be 8745 (95% PI 8558, 8933), 
of which our model predicted 67.9% (n=5939; 95% PI 
5785, 6102) would be seen through EIS. Only 176 (95% 
PI 151, 203) cases aged 16-64 years were forecast in 
Wales per annum. Assuming that our prediction model 
is accurate, it indicated that the Department of Health's 
current uniform rate of 51/100 000 person-years was 
higher than the predicted point estimates for rates fore- 
cast by our PsyMaptic model in 351 LADs (93%) in 
England and Wales, but was lower than that predicted by 
our model in Birmingham and several London bor- 
oughs (see online supplementary figure S2, left-hand 
map). Under a conservative approach, these differences 
achieved statistical significance in parts of London 
(where the Department of Health's model underesti- 
mated the need as predicted by PsyMaptic), and in 
some more rural parts of England and Wales (where the 
Department of Health's model overestimated the need; 
see online supplementary figure S2, right-hand map) . 

DISCUSSION 
Principal findings 

We have developed and tested several epidemiological 
prediction models to forecast FEP incidence in England 
and Wales, having taken into account regional differ- 
ences in the sociodemographic and socioenvironmental 
profiles of different populations. Inspection of our data 
suggested that a model fitted with age group, sex, their 
interaction, ethnic group and LAD-level population 
density provided the greatest external predictive validity 
when compared with the observed FEP caseload ascer- 
tained through EIS in our validation sample. This model 
also had good apparent validity across the entire age 
range (16-64 years). All models outperformed the 
Department of Health's current gold standard for EIS 
commissioning,^ based on a uniform incidence rate. 
Our data suggested that the original figure used to com- 
mission EIS probably overestimated the true incidence 
of FEP in rural areas, and underestimated rates in urban 
settings. However, we acknowledge that commissioning 
decisions will need to be based on several additional 
factors, including the level of preclinical or non- 
psychotic psychopathology requiring assessment at initial 
referral to EIS, and variation in service organisation, 
remit and delivery. 

Limitations and future development 

Our prediction models were based on epidemiological 
data obtained from large, robust population-based FEP 
studies for people aged 16-64 years. The best-fitting 
model had good apparent validity over this age range. 
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Table 3 Observed versus predicted cases in Social Epidemiology of Psychoses in East Anglia study for all clinically relevant psychoses, 16-35 years* 







Predicted (95% PI) 


Predicted (95% PI) 


Predicted (95% PI) 


Predicted (95% PI) 


EIS 


Observed 


IVIodel 1 


Model 2 


Model 3 


Model 4 


Overall total, 16-35 years 


522 


641.2 (586.0, 696.1) 


468.5 (422.0, 518.0) 


474.7(429.0, 522.0) 


487.5(441.0, 535.0) 


CAMEO North 


55 


84.7 (66.0, 106.0) 


68.5 (52.0, 87.0) 


67.8 (51.0, 86.0) 


70.2 (53.0, 90.0) 


CAMEO South 


134 


163.6 (135.0, 192.0) 


105.5 (84.0, 129.0) 


111.7(89.0, 134.0) 


110.9 (90.0, 132.0) 


1 West Norfolk 


26 


29.2 (18 A 41.0) 


23.0 (13.0, 35.0) 


22.0 (12.0, 32.0) 


23.4 (14.0, 34.0) 


W Central Norfolk 


120 


162.6 (136.0, 191.0) 


122.6 (99.0, 148.0) 


123.0 (100.0, 149.0) 


128.3 (104.0, 152.0) 


Great Yarmouth and Waveney 


60 


49.1 (34.0, 65.0) 


41.6 (28.0, 55.0) 


39.9 (27.0, 53.0) 


42.9 (30.0, 57.0) 


Suffolk 


127 


151.9(126.0, 178.0) 


107.4 (85.0, 128.0) 


110.3(88.0, 133.0) 


111.7(90.0, 136.0) 


Overall total, 36-64 years 




332.2 (292.0, 373.0) 


244.0 (213.0, 276.0) 


248.6 (216.0, 280.0) 


256.3 (228.0, 291 .0) 






IVIodel 5 


Model 6 


Model 7 




Overall total, 16-35 years 


522 


477.1 (428.0, 523.0) 


508.5 (459.0, 559.0) 


715.6 (664.0, 769.0) 




CAMEO North 


55 


69.1 (52.0, 88.0) 


64.5 (48.0, 82.0) 


92.1 (74.0, 111.0) 




CAMEO South 


134 


100.7 (79.0, 123.0) 


131.1 (108.0, 157.0) 


169.0 (144.0, 195.0) 




West Norfolk 


26 


24.7(16.0, 35.0) 


22.3 (13.0, 33.0) 


35.8 (25.0, 48.0) 




1 Central Norfolk 


120 


128.2(105.0, 153.0) 


132.5 (108.0, 157.0) 


187.7 (161.0, 215.0) 




P Great Yarmouth and Waveney 


60 


47.7(35.0, 63.0) 


38.0 (26.0, 51.0) 


59.1 (44.0, 74.0) 




Suffolk 


127 


106.6 (86.0, 128.0) 


120.0 (96.0, 143.0) 


172.1 (147.0, 198.0) 




Overall total, 36-64 years 




249.7 (221 .0, 284.0) 


262.9 (233.0, 297.0) 


1175.4 (1109.0, 1243.0) 





*Numbers in italics denote where the observed count fell within 95% prediction interval (95% PI) for people aged 16-35 years. Observed data for people aged 36-64 years in the validation 
sample were not available. 

Model 1: Age group, sex, their interaction and ethnicity. 

Model 2: Model 1+IMD. 

Model 3: Model 1+extent of deprivation. 

Model 4: Model 1+income deprivation. 

Model 5: Model 1 +employment deprivation. 

Model 6: Model 1+population density. 

Model 7: Department of health uniform figure for EIS of 15 new cases per ICQ 000 people/year. 
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Table 4 External model validation diagnostics* 



IVIodel 



Observed case count within 
SEPEA overall prediction 
intervals? (rank) 



EIS (N=6) 



LAD (N=21) 



Number 
correct (rank) 



RMSE 
(rank) 



Number 
correct (rank) 



RMSE 
(rank) 



Mean ranking (rank 
of mean ranking) 



Model 1 
Model 2 
Model 3 
Model 4 
Model 5 
Model 6 
Model 7 



No (5) 
No (5) 
Yes (1) 
Yes (1) 
Yes (1) 
Yes (1) 
No (5) 



3(6) 
4(4) 
5(1) 
4(4) 
5(1) 
5(1) 
2(7) 



26.9 (6) 

17.0 (4) 

15.1 (2) 
15.1 (2) 
18.0 (5) 
11.6 (1) 
39.4 (7) 



18(2) 
18(2) 
17(5) 
17(5) 
18(2) 
19(1) 
13(7) 



8.9 (6) 
6.5 (4) 
6.2 (3) 
6.1 (1) 
6.7 (5) 
6.1 (1) 
11.7 (7) 



5.0 (6) 
4.8 (5) 
2.4 (2) 
2.6 (3) 
2.8 (4) 
1.0 (1) 
6.6 (7) 



*For each diagnostic, models are placed in rank order (1=best model, 7=worst model) with ties given the same ranking. The mean ranking 
and rank provide an estimate of the overall performance of various models. 

EIS, early intervention services; LAD, local authority district; RMSE, root mean squared error; SEPEA, Social Epidemiology of Psychoses in 
East Anglia. 



and good external validity over the age range 16-35 
years. While 16-35 years covers the majority of adult 
onset psychosis cases seen in mental health services, we 
recognise that some EIS teams incept people from 
14 years old. We were unable to extrapolate our models 
to this age range, given the current absence of incidence 
data for this group in England. Data from Scandinavia 
suggest that the incidence of such 'early onset' psychoses 
is absolutely low,^^ although the rate may have increased 
over the last few decades, probably as a result of move- 
ment towards earlier detection. We were also unable to 
externally validate prediction models for people aged 
36-64 years, because comparable observed incidence 
data were not available in our validation sample. We have 
no reason to believe our predictions will be invalid for 
this group, however, since the empirical data which 
underpinned our models were ascertained from the 
same two large, well-conducted studies as for data on 

18 19 28 

the younger age group. Furthermore, published 

findings from these studies are consistent with the wider 
epidemiological literature on FEP in England and inter- 

1*7 21 30 

nationally It will be important to validate the pre- 

dictive capability of our model (s) in this age range, and 
we will seek to identify suitable samples to do so in 
future versions of PsyMaptic. 

Our best-fitting overall model demonstrated excellent 
external validity for predicting sex-specific FEP cases in 
our validation region (ie, SEPEA) . It performed less well 
across age-specific and ethnic-specific stratum in this 
region. With respect to age, this discrepancy is most 
likely to be a function of EIS provision itself, which seeks 
to intervene as early as possible in the onset of psychosis. 
The effect of this will reduce median age at onset in 
comparison to studies conducted prior to the introduc- 
tion of EIS, such as the yESOP and ELFEP studies upon 
which our models are based. Future versions of 
PsyMaptic will incorporate empirical data from post-EIS 
studies to improve age-specific predictions. The validity 
of our model in some ethnic groups also requires 
further refinement. Much of the prediction data under- 
lying our models came from urban environments with 



large proportions of ethnic minority groups. The socio- 
demographic profile and sociocultural experiences of 
these groups may be very different to those of their 
counterparts in other, less urban, parts of England, thus 
altering psychosis risk in different ethnic groups. In our 
observed data, a larger proportion of cases were white 
British than predicted by our model. If ethnicity is a 
partial proxy for exposure to deleterious socioenviron- 
mental experiences, such as the combined effect of 
social inequality, fragmentation, deprivation and popula- 
tion density,^^ then simultaneously incorporating such 
factors into our models may improve their predictive val- 
idity by ethnicity. Alternatively, risk by ethnic group may 
be conditional upon (ie, interact with) environmental 
factors in urban areas (as with the ethnic density 
effect^^ ^^), but whether such interactions exist in less 
urban regions is not known. Forthcoming SEPEA and 
PsyMaptic data will explore such possibilities. 

All prediction models had reasonable apparent valid- 
ity, although our proposed model performed slightly 
worse (most noticeably for AIC) than models which 
included deprivation (ie, models 2-4) instead of popula- 
tion density. Our decision to use model 6 as our pro- 
posed candidate for the prediction tool was supported 
by the fact that it produced the most accurate external 
forecasts of any model, despite considerable 
socioenvironmental differences between regions in our 
prediction and validation samples. We were unable to 
predict the expected incidence of psychotic disorder in 
geographical areas smaller than LADs, such as electoral 
wards, or to other parts of the UK, because appropriate 
denominator data were not published as mid-term 
census estimates. The 2011 census will provide small 
area and national data for the whole of the UK, sched- 
uled for release in mid-2013. This will allow us to update 
our tool to the latest population estimates for the UK, 
and refine our PsyMaptic tool at a smaller geographical 
level for fine-grained healthcare commissioning. We will 
then be able to develop models to explore cross-level 
interactions, such as the association between individual 
ethnicity and neighbourhood-level ethnic density. Small 
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area prediction models will require a multilevel 
approach, not attempted here, because obtaining pre- 
dictions from multilevel random effects models is not 
straightforward and requires active statistical 
development. 

We believe case ascertainment in our validation sample 
led to a reliable estimate of the incidence of psychotic 
disorder for people aged 16-35 years. EIS were the only 
mental health service for people aged 14-35 years 
experiencing a first episode of psychosis in East Anglia, 
minimising the potential for underascertainment in the 
population at-risk when derived from careful epidemio- 
logical design. We are confident that our validation 
sample also contained few false positive cases for any clin- 
ically relevant psychoses, since participants were 
excluded who failed to meet acceptance criteria for EIS, 
or who did not meet clinical diagnosis for psychotic dis- 
order in the first 6 months following EIS acceptance. It 
is important to recognise that while our prediction 
models are based on diagnosed clinically relevant psych- 
otic disorders, service commissioning will also need to 
account for additional preclinical or non-psychotic psy- 
chiatric morbidity presenting to EIS, particularly in ser- 
vices which operate early detection models or 
implement 'watch-and-wait' briefs. The SEPEA data used 
to validate our models do not predict (1) the number of 
'false positive' subjects who may require psychiatric 
triage and assessment, even though they are not 
accepted by EIS or (2) the number of 'true positive' sub- 
jects accepted by services, but who did not meet epi- 
demiological criteria for inclusion in the validation 
sample of the SEPEA study (ie those living outside the 
catchment area at first contact, or those transferred 
from other services); these people will consume varying 
degrees of service resources which need to be consid- 
ered in service planning. 

We also note that pathways to care may affect the level 
of incidence observed in EIS, since many filters are 
likely to operate before subjects come to the attention of 
EIS. These will include local level service organisation 
and the relationship between Community Mental Health 
Teams, Child and Adolescent Mental Health and EIS. 
Furthermore, acceptance criteria for entry to EIS vary, 
which will have a downstream effect on the number of 
new cases of clinically relevant psychoses received in 
each team. Future versions of PsyMaptic will include 
forecasts for specific psychotic disorders, as standardised 
research-based diagnoses (using OPCRIT^^) are cur- 
rently being collected in the ongoing SEPEA study. 
Acceptance rates to EIS may also be influenced by local 
community awareness of such services. While our predic- 
tion models outperformed the current gold standard for 
EIS commissioning in England when restricted to clinic- 
ally relevant caseloads, we recommend that our models 
are best interpreted as forecasts of the expected burden 
of first episode psychosis in given populations, not the 
total burden of resource consumption through EIS, 
given these issues. 



We estimated Pis from first principles (DJ) since their 
derivation is an area of statistical development.^^ We 
used a bootstrap-like methodology to produce 95% PI 
accounting for natural variation in the validation sample, 
but ignoring parameter uncertainty in the coefficients 
included in prediction models, which we assumed to be 
the true coefficients of risk in the population. Our 
approach therefore naturally led to slightly artificially 
narrow 95% Pis. This was not necessarily undesirable for 
the purpose of model validation and the precise predic- 
tion of expected counts, because we wished to apply 
stringent criteria. Ideally, 95% Pis should take into 
account both these sources of variation, although we 
note that parameter uncertainty is usually small com- 
pared with the natural variation of the quantities of 
interest. The addition of more empirical data in the pre- 
diction sample would not lead to narrower 95% Pis, 
though it would tend to move the point estimate of risk 
for each coefficient closer to the true value in the popu- 
lation. We do not believe we have misestimated the point 
estimates of risk across major sociodemographic groups, 

1 V 21 22 

since our results accord with the wider literature. 
We sought independent confirmation that our develop- 
ment of 95% PI was correct (personal communication 
with Professor Ian White, MRC Biostatistics Unit). We 
recommend that all prediction point estimates from our 
PsyMaptic model are considered with their 95% Pis, 
which provide information about the natural variance in 
expected rates in the population. 

IVIeaning of the findings 

If commissioners are to meet the Department of 
Health's vision to orientate health services around local 
need,^ ^ ^ differences in the demand for EIS and other 
mental and physical health services will need to be 
taken into account to allocate finite resources where 
they are most needed. The PsyMaptic prediction model 
provides proof-of-concept that when robust empirical 
epidemiological data are combined with accurate popu- 
lation at-risk estimates, this can be realised. As such, our 
modelling approach could have utility in many other set- 
tings and for many disorders. Our translational 
approach demonstrated good validity to predict the 
expected incidence of first episode psychosis, particu- 
larly through EIS, where 76% and 63% of all male and 
female adult-onset FEP cases, respectively, will typically 
present. Since their inception in 2002, EIS in England 
and Wales have reported both lower^^ and higher^^ case- 
loads than they were originally envisioned to manage,^ 
with shortfalls or excesses in anticipated demand for ser- 
vices aligned to the degree of urbanisation in the under- 
lying catchment area. Others have noted that EIS 
provision in rural areas may be difficult to implement 
effectively,^"^ and while the MH-PIG acknowledged 
that "...(a)n understanding of local epidemiology is 
needed as the size of population covered will depend on 
a number of different factors" (ref. 5, p. 55), no further 
elaboration on how to achieve this was provided. We 
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believe PsyMaptic provides a possible tool to overcome 
this challenge, improving the description and prediction 
of local population need beyond the MH-PIG and 
including individual-level and neighbourhood-level indi- 
cators of local need/^ From an aetiological perspective, 
we acknowledge that variables such as ethnicity or popu- 
lation density are likely to be markers for a suite of more 
complex, interactive social, genetic and environmental 
determinants of psychosis. 

Our models are not the first to be used to forecast 
mental illness needs in England and Wales, though we 
believe this is the first attempt to forecast incidence rather 
than prevalence in the community. We recommend that 
our prediction methodology is used in conjunction with 
the wide range of public health observatory data avail- 
able,^^ as well as the caveats presented above. PsyMaptic 
has been included with other indicators in the Joint 
Commissioning Panel for Mental Health's forthcoming 
guidance for commissioning of public mental health ser- 
vices.^^ Ongoing monitoring and audit of EIS will be vital 
to ensure that services meet the fidelity criteria upon 
which they were originally commissioned,^^ including 
ensuring that service capacity matches local need as 
closely as possible. As part of this process, we will need to 
externally validate our models in a wider range of settings, 
refining them based on empirical observation. 

We note that advocacy expressed for EIS by healthcare 
professionals in England and Wales broadly correlates 
with demand for services as predicted by PsyMaptic. 
Though by no means universal, proponents of EIS tend 
to be located in major conurbations — such as London,^^ 
Birmingham"^^ or Manchester'^ — ^where the demand 
for EIS will be highest, while those who suggest EIS 
resources could be used more effectively through other 
types of mental health service provision tend to work in 
more rural communities,^^ where but a handful of 
young people would be expected to come to the atten- 
tion of services each year. It is possible that both sides 
are correct and that more resources are required to help 
with the tide of psychotic illness in inner cities. 
Resources might be used more effectively in other ways, 
elsewhere, so long as the needs of the small number of 
young people who suffer an FEP each year are met; a 
dedicated specialist EIS may not be the most effective 
approach when anticipated demand will be very low. 

Given the significant downstream economic savings 
associated with spending on EIS as estimated in an 
urban setting,^ PsyMaptic could be used to highlight 
regions where sufficient investment to appropriate 
mental health services would lead to the greatest eco- 
nomic gains in terms of mental healthcare expenditure 
(assuming sustained intervention also leads to improved 
social and clinical benefit for patients^ ^) . PsyMaptic can 
also be used to highlight regional variation in demand 
according to age and sex and, in future versions, by eth- 
nicity. This will allow service planners to tailor provision 
around the sociocultural characteristics of their local 



populations. Our prediction tool for first episode psych- 
osis, which translates robust empirical epidemiological 
data on psychosis risk to the population structure of dif- 
ferent regions, offers a methodology for improving the 
allocation of finite mental health resources based on 
local need. 
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